High Performance Scientific Computing in Applications with Direct Finite Element Simulation

To predict separated flow including stall of a full aircraft with Computational Fluid Dynamics (CFD) is considered one of the problems of the grand challenges to be solved by 2030, according to NASA [1]. The nonlinear Navier- Stokes equations provide the mathematical formulation for fluid flow in 3- dimensional spaces. However, classical solutions, existence, and uniqueness are still missing. Since brute-force computation is intractable, to perform predictive simulation for a full aircraft, one can use Direct Numerical Simulation (DNS); however, it is prohibitively expensive as it needs to resolve the turbulent scales of order Re4 . Considering other methods such as statistical average Reynolds’s Average Navier Stokes (RANS), spatial average Large Eddy Simulation (LES), and hybrid Detached Eddy Simulation (DES), which require less number of degrees of freedom. All of these methods have to be tuned to benchmark problems, and moreover, near the walls, the mesh has to be very fine to resolve boundary layers (which means the computational cost is very expensive). Above all, the results are sensitive to, e.g. explicit parameters in the method, the mesh, etc. As a resolution to the challenge, here we present the adaptive time- resolved Direct FEM Solution (DFS) methodology with numerical tripping, as a predictive, parameter-free family of methods for turbulent flow. We solved the JAXA Standard Model (JSM) aircraft model at realistic Reynolds number, presented as part of the High Lift Prediction Workshop 3. We predicted lift Cl within 5% error vs. experiment, drag Cd within 10% error and stall 1◦ within the angle of attack. The workshop identified a likely experimental error of order 10% for the drag results. The simulation is 10 times faster and cheaper when compared to traditional or existing CFD approaches. The efficiency mainly comes from the slip boundary condition that allows coarse meshes near walls, goal-oriented adaptive error control that refines the mesh only where needed and large time steps using a Schur-type fixed-point iteration method, without compromising the accuracy of the simulation results. As a follow-up, we were invited to the Fifth High Order CFD Workshop, where the approach was validated for a tandem sphere problem (low Reynolds number turbulent flow) wherein a second sphere is placed a certain distance downstream from a first sphere. The results capture the expected slipstream phenomenon, with appx. 2% error. A comparison with the higher-order frameworks Nek500 and PyFR was done. The PyFR framework has demonstrated high effectiveness for GPUs with an unstructured mesh, which is a hard problem in this field. This is achieved by an explicit time-stepping approach. Our study showed that our large time step approach enabled appx. 3 orders of magnitude larger time steps than the explicit time steps in PyFR, which made our method more effective for solving the whole problem. We also presented a generalization of DFS to variable density and validated against the well-established MARIN benchmark problem. The results show good agreement with experimental results in the form of pressure sensors. Later, we used this methodology to solve two applications in multiphase flow problems. One has to do with a flash rainwater storage tank (Bilbao water consortium), and the second is about designing a nozzle for 3D printing. In the flash rainwater storage tank, we predicted that the water height in the tank has a significant influence on how the flow behaves downstream of the tank door (valve). For the 3D printing, we developed an efficient design with the focused jet flow to prevent oxidation and heating at the tip of the nozzle during a melting process. Finally, we presented here the parallelism on multiple GPUs and the embedded system Kalray architecture. Almost all supercomputers today have heterogeneous architectures, such as CPU+GPU or other accelerators, and it is, therefore, essential to develop computational frameworks to take advantage of them. For multiple GPUs, we developed a stencil computation, applied to geological folds simulation. We explored halo computation and used CUDA streams to optimize computation and communication time. The resulting performance gain was 23% for four GPUs with Fermi architecture, and the corresponding improvement obtained on four Kepler GPUs were 47%. The Kalray architecture is designed to have low energy consumption. Here we tested the Jacobi method with different communication strategies. Additionally, visualization is a crucial area when we do scientific simulations. We developed an automated visualization framework, where we could see that task parallelization is more than 10 times faster than data parallelization. We have also used our DFS in the cloud computing setting to validate the simulation against the local cluster simulation. Finally, we recommend the easy pre-processing tool to support DFS simulation.La Caixa 201

Direct FEM computation of turbulent multiphase flow in 3D priting nozzle design

In this paper, we present a nozzle design of the 3D printing using FEniCS-HPC as mathematical and simulation tool. In recent years 3D printing or Additive Manufacturing (AM) has become a emerging technology and it has been already in use for many industries. 3D printing considered as a sustainable production or eco-friendly production, where one can minimize the wastage of the material during the production. Many industries are replacing their traditional parts or product manufacturing into optimized or smart 3D printing technology. In order to have 3D printing to be efficient, this should have optimized nozzle design. Here we design the nozzle for the titanium material. Since it is a metal during the process it has to be preserved by the inert gas. All this makes this problem comes under the multiphase flow. FEniCS-HPC is high level mathematical tool, where one can easily modify a mathematical equations according to the physics and has a good scalability on massively super computer architecture. And this problem modelled as Direct FEM/General Galerkin methodology for turbulent incompressible variable-density flow in FEniCS-HP

Direct FEM large scale computation of turbulent multiphase flow in urban water systems and marine energy

High-Reynolds number turbulent incompressible multiphase flow represents a large class of engineering problems of key relevance to society. Here we describe our work on modeling two such problems: 1. The Consorcio de Aguas Bilbao Bizkaia is constructing a new storm tank system with an automatic cleaning system, based on periodically pushing tank water out in a tunnel 2. In the framework of the collaboration between BCAM - Basque Center for Applied Mathematics and Tecnalia R & I, the interaction of the sea flow with a semi submersible floating offshore wind platform is computationally investigated. We study the MARIA' benchmark modeling breaking waves over objects in marine environments. Both of these problems are modeled in the the Direct FEM/General Galerkin methodology for turbulent incompressible variable-densitv flow 1,2 in the FEniCS software framework

Towards HPC-Embedded Case Study: Kalray and Message-Passing on NoC

Today one of the most important challenges in HPC is the development of computers with a low power consumption. In this context, recently, new embedded many-core systems have emerged. One of them is Kalray. Unlike other many-core architectures, Kalray is not a co-processor (self-hosted). One interesting feature of the Kalray architecture is the Network on Chip (NoC) connection. Habitually, the communication in many-core architectures is carried out via shared memory. However, in Kalray, the communication among processing elements can also be via Message-Passing on the NoC. One of the main motivations of this work is to present the main constraints to deal with the Kalray architecture. In particular, we focused on memory management and communication. We assess the use of NoC and shared memory on Kalray. Unlike shared memory, the implementation of Message-Passing on NoC is not transparent from programmer point of view. The synchronization among processing elements and NoC is other of the challenges to deal with in the Karlay processor. Although the synchronization using Message-Passing is more complex and consuming time than using shared memory, we obtain an overall speedup close to 6 when using Message-Passing on NoC with respect to the use of shared memory. Additionally, we have measured the power consumption of both approaches. Despite of being faster, the use of NoC presents a higher power consumption with respect to the approach that exploits shared memory. This additional consumption in Watts is about a 50%. However, the reduction in time by using NoC has an important impact on the overall power consumption as well

Living with multiple myeloma: A focus group study of unmet needs and preferences for survivorship care

Purpose: To describe the unmet informational, psychological, emotional, social, practical, and physical needs and preferences for posttreatment survivorship care of individuals living with multiple myeloma to inform the development of relevant, personcentered, survivorship services. Methods: An exploratory, descriptive study using 2 focus groups with 14 participants, 6 to 49 months postdiagnosis. Results: Thematic analysis revealed 7 key themes: information needs, experience with health-care professionals, coping with side effects, communicating with family and friends, dealing with emotions, support needs, and living with the chronicity of myeloma. Participants described key characteristics of survivorship care relevant to their needs and indicated they would like a more whole of person approach to follow-up when the main treatment phases had completed. Conclusion: Participants in this study described unmet needs across a breadth of domains that varied over time. The development of flexible, person-centered approaches to comprehensive survivorship care is needed to address the considerable quality-of-life issues experienced by people living with multiple myeloma. Nurse-led care may offer 1 viable model to deliver enhanced patient experience—providing the vital “link” that people described as missing from their survivorship care

Time-resolved Adaptive Direct FEM Simulation of High-lift Aircraft Configurations

Our simulation methodology is referred to as Direct FEM Simulation (DFS), or General Galerkin (G2) and uses a finite element method (FEM) with piecewise linear approximation in space and time, and with numerical stabilization in the form of a weighted least squares method based on the residual. The incompressible Navier-Stokes Equations (NSE) are discretized directly, without applying any filter. Thus, the method does not result in Large Eddy Simulation (LES) filtered solutions, but is instead an approximation of a weak solution satisfying the weak form of the NSE. In G2 we have a posteriori error estimates for quantities of interest that can be expressed as functionals of a weak solution. These a posteriori error estimates, which form the basis for our adaptive mesh refinement algorithm, are based on the solution of an associated adjoint problem with a goal quantity (the aerodynamic forces in this work) as data, similarly to an optimal control problem. We provide references to related work below. The methodology and software have been previously validated for a number of turbulent flow benchmark problems, including one of the HiLiftPW-2 high Reynolds number cases. The DFS method is implemented in the Unicorn solver, which uses the open source software framework FEniCS-HPC, designed for automated solution of partial differential equations on massively parallel architectures using the FEM. In this chapter we present adaptive results from the Third AIAA High Lift Prediction Workshop in Denver, Colorado based on our DFS methodology and Unicorn/FEniCS-HPC software. We show that the methodology quantitavely and qualitatively captures the main features of the experiment - aerodynamic forces and the stall mechanism with a novel numerical tripping, with a much coarser mesh resolution and cheaper computational cost than the standard in the field

On the Specificity of Heparin/Heparan Sulfate Binding to Proteins. Anion-Binding Sites on Antithrombin and Thrombin Are Fundamentally Different

Background The antithrombin–heparin/heparan sulfate (H/HS) and thrombin–H/HS interactions are recognized as prototypic specific and non-specific glycosaminoglycan (GAG)–protein interactions, respectively. The fundamental structural basis for the origin of specificity, or lack thereof, in these interactions remains unclear. The availability of multiple co-crystal structures facilitates a structural analysis that challenges the long-held belief that the GAG binding sites in antithrombin and thrombin are essentially similar with high solvent exposure and shallow surface characteristics. Methodology Analyses of solvent accessibility and exposed surface areas, gyrational mobility, symmetry, cavity shape/size, conserved water molecules and crystallographic parameters were performed for 12 X-ray structures, which include 12 thrombin and 16 antithrombin chains. Novel calculations are described for gyrational mobility and prediction of water loci and conservation. Results The solvent accessibilities and gyrational mobilities of arginines and lysines in the binding sites of the two proteins reveal sharp contrasts. The distribution of positive charges shows considerable asymmetry in antithrombin, but substantial symmetry for thrombin. Cavity analyses suggest the presence of a reasonably sized bifurcated cavity in antithrombin that facilitates a firm ‘hand-shake’ with H/HS, but with thrombin, a weaker ‘high-five’. Tightly bound water molecules were predicted to be localized in the pentasaccharide binding pocket of antithrombin, but absent in thrombin. Together, these differences in the binding sites explain the major H/HS recognition characteristics of the two prototypic proteins, thus affording an explanation of the specificity of binding. This provides a foundation for understanding specificity of interaction at an atomic level, which will greatly aid the design of natural or synthetic H/HS sequences that target proteins in a specific manner

Finite Element Simulations of Two-phase Flow and Floating Bodies Using FEniCS-HPC

We present a variational multiscale stabilized finite element method to solve the variable density incompressible Navier-Stokes equations for the simulation of two-phase flow. We introduce a level-set method based on the compression technique similar to [1]. For the simulation of floating devices we make use of a simplified rigid body motion scheme and a deforming mesh approach [2]. The mesh deforms elastically following the movement of the body. An implicit turbulence model is used where turbulence is modelled by the numerical stabilization. The described methods are implemented in the open source software framework FEniCS-HPC [3] provided with an automated methodology for discretization and error control. We are working in a project for marine energy generation together with Tecnalia R&I. In this context we simulate floating platforms that will be used for marine energy generation or device experimentation in the ocean. The aim is to study the dynamics of this kind of off-shore devices. Our simulation results are compared against the experimental data obtained by Tecnalia R&I company in the experimental tank of CEHIPAR in Spain. We also participate in the IEA-OES Task 10 project where different simulations of floating bodies are carried out. The results are compared against other groups simulations that use different methodologies